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Abstract Body 



Background / Context: 

A large proportion of the high school graduates who attend a postsecondary institution 
take at least one basic skills/developmental/remedial course in either mathematics or English 
(Parsad & Lewis, 2003). A report from the Academic Senate for California Community Colleges 
(2000), which surveyed almost half of the community colleges in the state, found that, on 
average, 49 percent of students are directed to these basic skills courses. 

There is considerable debate on the effects and benefits of remediation in community 
colleges. Proponents argue that it enables poorly prepared high school students to attain the 
necessary preparation to succeed in college (Boylan, Bliss, & Bonham, 1994; 1997; Lazarick, 
1997). On the other hand critics argue that the benefits of remediation are not clear given there is 
no evidence that remediated students passed college-level courses or attained degrees at higher 
rates than non-remediated students (Calcagno, 2007; Calcagno & Long, 2008; Martorell & 
McFarlin, 2007). In addition, there is evidence that students who were placed in remediation but 
who took college credit courses passed these courses (Armstrong, 1999). Finally, a major 
criticism is that, relative to its potential benefits, remediation is too costly for the students and for 
the state (James, Morrow & Perry, 2002). 

Every year more than 50 percent of the students from the LACCD are placed into basic 
skills mathematics. This is costly for the individuals and for the state. At the individual level, 
students need to take these courses before they can enroll in degree or transfer level courses, and 
this substantially increase the time that they remain in the system. The state in turn has to pay for 
courses that the students were supposed to take in high school. As described below, the effects of 
assignment of entering students to different levels of math on the academic preparation, 
persistence and educational outcomes are unclear at best. This study is an ambitious attempt to 
enhance the available research on this critical policy issue for the State of California and the 
nation. This evaluation will provide insights to the district and the state about the effect of 
placement on successful course sequences that have the potential to promote persistence and 
save millions of dollars to the state. 

Purpose / Objective / Research Question / Focus of Study: 

The main objective of our proposed study is to evaluate the effectiveness of math 
placement policies for entering community college students on these students’ academic success 
in math, and their transfer and graduation rates. The main research question that guides the 
proposed study is: What are the effects of various basic skills mathematics paths on the course- 
taking patterns of community college students? Our study combines descriptive analyses with a 
regression discontinuity (RD) design to estimate the effect of developmental education. We have 
two closely related goals: 1) To provide a descriptive analysis of the course taking patterns (basic 
skills and college level) of community college students; 2) To use a regression-discontinuity 
design to test the effect of assignment to different levels of basic skills courses in mathematics on 
subsequent educational outcomes. Our outcomes of interest all relate to the students’ educational 
attainment: (i) successfully passed to the next math level; (ii) enrolled and passed a college level 
course; (iii) community college grade point average; (iv) total college credit attained (total and 
transfer level); (v) attained a certificate or associate degree; and (vi) transferred to a four-year 
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college. We chose a RD approach because this is a technique that enables the researcher to make 
causal inferences when randomization is not feasible. The general idea is that the researcher 
“assigns” individuals to the treatment and control groups according to an exogenously 
detennined cutoff score on the assignment variable (i.e. the cutoff point on the placement test 
that they need to take their first year) rather than the fair coin toss that is common on randomized 
trials. We will use a sharp regression discontinuity (SRD) to estimate the intent to treat (ITT), 
and a fuzzy regression discontinuity (FRD) to estimate the local average treatment effect 
(LATE). 

The study described above is currently being funded by an Institute of Education 
Sciences (IES) research grant. At this stage the authors are still in the process of creating the 
final dataset, and of identifying the assessment and placement policies in the 9 community 
colleges. The authors expect that by the time of the conference next year, they would have 
complete infonnation regarding the placement rules for the 9 community colleges of the district. 
The authors will present an overview of the study, a brief summary of the literature, and they 
will present the descriptive statistics, as well as the placement policies in each of the 9 colleges. 

Setting: 

Description of the research location. 

The setting in which the study is conducted is the Los Angeles Community College 
District (LACCD). We use student transcript data since 2001 until 2009. For this first stage of 
the project we use the transcript data to evaluate the effect of taking a basic skills mathematics 
course on the probability of passing either the following level of basic skills math, or the next 
college level course. 

Population / Participants / Subjects: 

The sampling frame for our analysis consists of about 158,000 community college 
students who were placed into mathematics in one of nine Los Angeles County Community 
Colleges between June 2001 and September 2009 and for whom a Continuous Placement Score 
(CPS) is available in the management infonnation system maintained by the Los Angeles 
Community College District. This includes about 50 percent of all entering students at these nine 
colleges. For each of these students, we will gain access to at least two years of course transcript 
data, including whether the student attained a certificate or an associate degree, transferred, or 
left college before graduation. The length of follow up of the outcomes will vary depending on 
the entering year. To assess the sensitivity of our analyses to censoring, we will compare 
estimates from early cohorts to estimates from later cohorts. 



Intervention / Program / Practice: 

Description of the intervention, program or practice, including details of administration and duration. 

Assessment in mathematics in the Los Angeles Community College District 
The results of the recent survey of the task force on assessment as well as infonnal 
conversations with placement administrators in the nine colleges of the LACCD indicate that 
there is a great deal of consistency in the assessment practices for placement in mathematics, 
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especially compared to English and ESL. Currently, the majority of colleges use one of the two 
commercially developed tests. Specifically, six colleges use the Accuplacer test, two use the 
Compass test, and one uses the MDTP. According to assessment staff, the test scores are used to 
place students, and these placement decisions are consistently enforced. In other words, students 
whose test scores indicate a need to take an elementary algebra course cannot enroll in any 
college-level math courses. Logistically, it is difficult for students to retake a placement exam to 
improve their scores. The colleges have specific dates and preparation scheduled for the 
placement exams and the colleges have to pay for each student who takes one of the 
commercially developed tests, making them unlikely to encourage or allow students to take a 
placement test more than once. 

Research Design: 

As introduced above, the study addresses these research questions using a regression 
discontinuity design. Such a design is the closest non-experimental research design to a random 
assignment experiment in which a portion of entering students would be assigned to one level of 
math and a portion would be assigned to the next higher level. The regression discontinuity 
design requires (a) a continuous predictor variable, and (b) discrete and exogenously established 
cut points. The latter must have a very strong relationship to the policy variable of interest. In the 
present study, the continuous predictor variable is the so-called “Continuous Placement Score” 
(CPS), which is derived from a college entry test that the vast majority of entering community 
college students take, as detailed in a previous section of this proposal. Each California 
community college chooses distinct cut scores on this CPS to place students in 4-5 different 
levels of math. While these referrals are not the only determinant of students’ subsequent 
enrollment decisions, they are a critical part of these decisions. 

As is the case in many true randomized experiments, the cut score in a regression 
discontinuity design merely identifies an “intent to treat.” Some proportion of sample members 
will not follow the researchers’ (or program) directions and the outcomes for those sample 
members are averaged in with outcomes for sample members who do act in accordance with the 
intended treatment. Thus, in the case of this particular study, a small proportion of the students 
will enroll in a different level of math than the placement score indicates, but their outcomes will 
be combined with those who do enroll in the assigned level to determine the impacts of 
assignment to different levels of math on the outcomes specified in the research questions above. 
This caveat must be considered when interpreting the results from our proposed research. In 
addition, and as we explain in more detail below we will estimate the models using fuzzy 
regression discontinuity in order to estimate the local average treatment effect (LATE) (Calcagno 
& Long, 2008). 

As discussed in more detail below, we are in the fortunate position that we have access to 
a very large sample of students. Across the nine institutions in the Los Angeles Community 
College district, we have access to transcript data for about 158,000 students across eight years. 
What this allows us to do is to estimate regression discontinuity functions for each of the four 
cut-points separating five levels of math instruction. The relatively short CPS score intervals 
between the cut points and the large sample of entering students cause the expected regression- 



A different way to conceptualize this issue is that the study will measure the impact of placement policies, not the 
impact of actual math instruction at different levels. To the extent that there is a great degree of overlap between the 
two, the results from the former can be generalized to the latter, but only with certain caveats. 
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discontinuity functions to be relatively “short and fat.” In other words, we will likely be able to 
limit our sample to students whose CPS scores are relatively close to the cutoff scores on either 
end. This has several important analytical advantages. First, it means that the functional form of 
the regression function is likely to be more straightforward than a long regression line covering a 
wide range of potential CPS scores would be. This in turn minimizes the likelihood that 
misspecification bias affects the impact estimates. The latter is one of the major risks to the 
internal validity of impact estimates obtained from regression discontinuity designs, especially 
compared with true randomized experiments, whose internal validity is, by definition, unaffected 
by specification error. Second, estimation of the regression function close to the cutoff score 
increases the policy relevance (external validity) of the impact estimates. 

Our study does have one additional complication, which is that the cut scores are 
independently detennined by the individual community colleges, so they will not be consistent 
across all nine, even though the CPS is comparable across all the community colleges. In 
California there is no consistent strategy to determine who needs remedial education. Each 
community college determines for itself which placement examinations to use and what the 
proficiency cut-scores will be for each test and placement decision. The community colleges in 
our sample use one of the following placement tests: the Accuplacer CPTs by the College Board, 
the Compass test developed by ACT, and the Math Diagnostic Testing Program (MDTP), 
originally developed by faculty from the California State University (CSU) and University of 
California (UC). 

Even though different tests are used to create the CPS and different cut scores are established 
by individual community colleges, the constructs will likely be sufficiently overlapping to allow 
analyses to be conducted pooled across all colleges, with appropriate interaction tenns to control 
for variation in the relationship of each college’s CPS with its students’ academic outcomes. We 
will also conduct separate impact analyses for each of the nine community colleges (or for 
groups of colleges that use similar tests and cut scores). 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

We recently received access to the different databases compiled by the LACCD. We are in the 
process of combining these datasets to have information related to individual student 
characteristics, course enrollment, and assessment and placement practices in a single database. 
In addition, we have devoted substantial amount of time to identifying the variables used in the 
assessment and placement. The fact that each college has a unique set of rules with different tests 
and cut scores to place students, and that this information has not been consistently used, means 
that we had to spend time identifying the different variables used in the placement from the 
databases. 

Findings / Results: 

Description of the main findings with specific details. 

We have been able to identify the assessment and placement practices in 4 of the 9 community 
colleges. Table 1 describes the placement rules for these 4 colleges. During the conference we 
will present an overview of the study, a brief literature review, and we will focus on describing 
how the placement rules result in the actual placement of students in the 9 community colleges. 
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Conclusions: 



California is going through a major financial crisis that has already reduced the funding 
for community colleges by 10 percent for the 2009-10 fiscal year. Colleges are already firing 
part-time faculty, increasing the number of students in already over-crowded class rooms, and 
reducing the number of pre-requisite courses (i.e. elementary algebra) for degree attainment. All 
of this is going to have a negative effect in the quality of the courses provided by colleges. After 
three years of a California Basic Skills Initiative that focused on professional development and 
that did not include any accountability measure it is imperative to fund a study that will illustrate 
the condition of math basic skills and its impact on short- and long-term educational outcomes. 
This study is an ambitious attempt to enhance the available research on this critical policy issue 
for the State of California and the nation. This evaluation will provide insights to the district and 
the state about the effect of placement on successful course sequences that have the potential to 
promote persistence and save millions of dollars to the state. 
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Appendix B. Tables and Figures 



Table l.A. Placement Rules for City College 














Multiple Measures Score Q 


MMP). Students can get up to 5 points based on their responses on the Educational Planning Questionnaire. 


Arithmetic (AR) 


Elementary Algebra (EA) 


College Level Math (CM) 






109 <= EA 


CM Referral* 










90<=EA< 109 


Math 125 (Inter Alg) 


103 < CM 


Math 261 (Calculus I) 






76 <= EA < 90 


Math 1 2 1 - 1 24 A (Elem Alg) 


87 < CM < 103 


Math 260 (Pre- 
Calculus) 






65 <= EA < 76 


MATH 115, 113, 112 (Elem 
Alg; 1 Sem) 


75 < CM < 87 


Math 230-240 (Math 
LA to Tng) 


65 <= AR 


EAREF 


50 < EA <65 


Math 113 (Elem Alg, 2 SEM) 


63 <= CM < 75 


Math 215-245 
(College Algebra) 


45<=AR<65 Math 112 


38<=EA<50 OR 
17<EA<38 &AR>65 


Math 112 (Pre-Alg) 


43 <= CM < 63 OR 
(17<CM<43 &EA>109) 


Math 125 (Inter Alg) 


17<=AR<45 Math 105 


17<=EA<38 


ARRef 


17 <= CM < 43 


EARef 


0<AR<17 


No ARP1 


0<EA<17 


No EA PI 


0 < CM < 17 


No CL PI 



Table l.B. Placement Rules for Southwest College 



Multiple Measures Score (MMP). Students can get up to 11 points and loose up to 3 points based 
on their responses on the Educational Planning Questionnaire. 



Arithmetic (AR) 


Elementary Algebra (EA) 


College Level Math (CM) 






74 >= EA 


CLM Ref 


104 < CM 


Above Math 240 


65 <= AR 


EA Ref 


70 < =EA <74 


Math 125 


86 <= CM < 104 


Math 240 


39<=AR<65 


Math 112 


41<=EA<70 " 


Math 115 


63 <= CM < 86 


M227-245 


22<=AR<39 


Math 105 


28<=EA<34 


Math 112 


43 <= CM < 63 


Math 125 


1<=AR<22 


Math 104 


0<=EA<28 


ARRef 


1 <= CM < 43 


EARef 
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Table l.C. Placement Rules for Tradetech College 


















Multiple Measures Score (MMP). No multiple measures used. 




Arithmetic (AR) 




Elementary Algebra (EA) 


College Level Math (CM) 














1113 <= AR<120 Math 115 






100 < CM 


Math 265 


100 <= AR< 113 


Math 113 


77 <= EA 120 


Math 125 


86 <= CM < 10' Math 260 


60<=AR<100 


Math 112 


34<=EA<77 


Math 115 


60 <= CM < 86 


Math 245 


22<=AR<60 


Math 105 


20<=EA<34 


Math 113 


40 <= CM < 63 


Math 125 


0<=AR<22 


Math 101 


0<=EA<20 


Math 112 


0 <= CM < 40 


Math 115 



Table l.D. Placement Rules for Tradetech College 
















Multiple Measures Score (MMP). Students can get up to 4 pomts based on 


their responses 


on the Educational Planning Questionnaire. 




Arithmetic (AR) 


Elementary Algebra (E.4 


College Level Math (CM) 


















74 <= EA 


CM Ref 


104 < CM 


Above Math 
240 


65 <= AR 


EA Ref 


70 <= EA <74 


Math 125 


86 <= CM < 104 


Math 240 


39<=AR<65 


Math 112 


4 1 <=EA<7 0 


Math 115 


63 <= CM < 86 


Math 245 


22<=AR<39 


Math 105 


28<=EA<34 


Math 112 


43 <= CM < 63 


Math 115 


1<=AR<22 


Math 104 


1<=EA<28 


ARRef 


1 <= CM < 43 


EA Ref 
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